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Summary of Review 

The School Choice Demonstration Project has published a series of reports written in the 
fifth and final year of its evaluation of the Milwaukee Parental Choice Program (MPCP). 
This review is of Report #32, which compares the test performance of MPCP students to 
that of a sample of students from the Milwaukee Public Schools (MPS). The comparisons 
revealed a mixed set of findings with no clear pattern. For example, a sample of low- 
income MPS students scored higher than MPCP students on average in 4th grade reading, 
math, and science, and in 8th and 10th grade math. MPCP students, on the other hand, 
scored higher than the MPS sample in 8th and 10th grade reading and science. The results 
are not particularly useful beyond providing a snapshot of how MPCP students and a 
comparison group of low-income MPS students performed on a battery of state exams. A 
supplemental analysis that found relatively more “very low” performing MPS schools was 
established by arbitrary cut scores and was potentially biased by unequal group sample 
sizes. The report correctly cautions readers not to make causal inferences about the effects 
of either sector given the descriptive nature of the analysis. 
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Review of 

SCDP Milwaukee Evaluation Report #32 

Casey D. Cobb, University of Connecticut 



I. Introduction 

The Milwaukee Parental Choice Program (MPCP) is currently the largest urban school 
voucher program in the nation. At present, nearly 21,000 students use a voucher of up to 
$6,442 to attend secular or religious private schools in Milwaukee. Voucher programs have 
long been the targets of intense debates over the efficient and appropriate use of public 
education funds. 

In February 2012 the School Choice Demonstration Project 1 released a series of final 
reports from its five-year evaluation of the MPCP. The Wisconsin Legislature in 2005 
required MPCP schools to administer nationally normed tests in grades 4, 8, and 10 to 
MPCP students, and also to submit the test scores to the School Choice Demonstration 
Project for purposes of evaluation. The reports address a range of issues related to the 
MPCP, including special education services, school climate and contexts, and test score 
performance. Of particular interest to policymakers has been the performance of MPCP 
students relative to students in the Milwaukee Public Schools (MPS). The School Choice 
Demonstration Project report under review here is Report # 32 , Milwaukee Longitudinal 
School Choice Evaluation: Annual School Testing Summary Report 2010 - 11 . It is 
authored by Anna M. J acob and Patrick J . Wolf 2 

II. Findings and Conclusions of the Report 

This report describes the test score performance of MPCP students on the 2010-2011 
Wisconsin Knowledge and Concepts Examinations (WKCE). The WKCE tests students in 
reading and math in grades 3-8 and 10, and tests students in science, language arts, 
writing, and social studies in grades 4, 8, and 10. The report compares MPCP results with 
those of a sample of MPS students. Because participation in the MPCP is restricted to low- 
income families, comparisons were made with MPS students who were eligible for federal 
free- and reduced- price lunch (FRL) subsidies. The comparisons revealed a mixed set of 
findings. For instance, FRL- MPS students scored higher than MPCP students on average 
in 4 th grade reading, math, and science, and in 8 th and 10 th grade math. MPCP students, on 
the other hand, scored higher on average than the FRL- MPS students in 8 th and 10 th grade 
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reading and science. Black- White and Latino- White achievement gaps were found to be 
larger among MPCP students than among low- income MPS students. All reported 
differences were large enough to be considered statistically significant. The report 
expressed differences in standard deviation units, which ranged from 0.06 to 0.95, with 
some favoring MPCP students and others favoring the low- income MPS students. 

The report offers the following conclusions: “In general, comparisons of the test-score 
performance of similarly disadvantaged students in the MPCP and MPS tend to favor the 
MPCP students in reading and science but the MPS students in math” (p. 25). Moreover, 
the “comparisons also tend to show MPCP students performing at lower levels than their 
MPS peers in 4 th grade but higher levels in 8 th and 10 th grades” (p. 25). 

One section of the report responds to “several media reports” that “have emphasized the 
fact that low-performing schools endure in the Choice Program and recommend that the 
public school system of accountability be more fully applied to schools in the MPCP” (p. 

19). The report concludes that school-level WKCE data “confirms that some MPCP schools, 
in some grades and some subjects, have very- low student performance . . . [and] also 
demonstrates that similarly low-performing schools endure in the MPS” (p. 19). Perhaps 
related to this point, the report’s executive summary notes a discrepancy between its 
results and results released by the Wisconsin Department of Public Instruction on March 
29, 2011. By law the Department was required to include students who did not take the test 
in its calculation of MPCP proficiency rates— thus counting them as non- proficient when in 
fact their proficiency was unknown (these non- testers would be part of the denominator in 
those computations). According to the report, this deflates proficiency rates among MPCP 
students relative to deflated rates among MPC students because more MPCP students (3%) 
than MPS students ( 1%) opted out of testing. As an alternative, the SCDP analysis excluded 
non- testers in the denominator. This alternative calculation helps readers see the various 
ways that test outcomes can be calculated and in turn how they can produce varying 
results. 

To be sure, the report emphasizes its descriptive nature (“this report describes the test 
scores of students in the two sectors, plain and simple”) (p. ii) and draws few direct 
inferences from the analysis. The descriptive data are meant to speak for themselves. 

III. The Report’s Rationale for Its Findings and Conclusions 

The report’s findings consist of descriptive statistics of test performance as measured by 
WKCE average scale scores and proficiency rates. Summary statistics are presented for 
MPCP and MPS sectors at the student and school levels. Histograms of scale scores from 
MPCP and MPS schools are also presented to explore differences in test score distributions 
across sectors. The report established a cut-score, scores below which signified “very low 
performance” (p. i). 3 No rationale was provided as to how these cut scores were chosen. 

The cut-scores were used to make the determination that “low-performing schools [at least 
equally] endure in the MPS under the public-school accountability system” (p. 19). The 
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report concludes that “[o]f the 76 very low performances by school, grade-level, and 
subject in 2010- 11, a total of 35 were from MPCP, while 41 were from MPS” (p. i). The 
report neglects to consider that the number of schools in the very- low- performance 
category could be a direct function of the number of schools in each sector sample. There 
are simply far more MPS schools in the sample (the report’s Table 5 on p. 14 shows that 
there are 44% more MPS schools in 4 th grade calculations, 24% more in 8 th grade, and 
143% more in 10 th grade). The probability is thus greater, assuming the two sector’s test 
scores are normally distributed, that more schools would fall in the tails of the distribution 
(see the report’s Figures 8ab and 9ab on p. 15). 

Throughout the report comparisons between MPCP and MPS students are qualified with 
apt warnings such as, “it would be a mistake for readers to draw conclusions concerning 
the effectiveness of the MPCP based on these or any other annual descriptive statistics” (p. 
4). The report’s authors are correct that the study design is not equipped for such causal 
inferences. The report offers no specific implications of the general patterns observed in 
the test performances of MPCP and low- income MPS students. The data are presented 
descriptively without much explanation. 

One of the report’s conclusions noted above (“comparisons also tend to show MPCP 
students performing at lower levels than their MPS peers in 4 th grade but higher levels in 
8 th and 10 th grades”[p. 25]) may slightly overstate the data. While the FRL-MPS sample did 
score higher on average than MPCP students in 4 th grade reading, math, and science, the 
8 th and 10 th grade MPCP students scored higher in reading and science, but not in math. 
MPCP and FRL-MPS students split on the staple disciplines of reading and math. 

IV. The Report’s Use of Research Literature 

No research literature was referenced in the report. There was really no need for it to do 
so, as this amounted to a descriptive report on test performance; no research questions 
were posed or explored. 

V. Review of the Report’s Methods 

The methods employed were sound, descriptive in nature, and for the most part clearly 
described. Expressing scale score differences in standard deviation units was particularly 
helpful, as was portraying the distribution of test scores among the sectors via histograms. 
The rationale for setting cut scores designated as representing “very low” test performance 
was not articulated. Further, the comparison between the numbers of “very low” 
performing schools in the MPCP and MPS sectors was potentially biased by unequal group 
sample sizes. The creation of the FRL-MPS sample was not ideal because the eligibility 
standards for federal FRL and the MPCP are different. 4 Nonetheless, using FRL represents 
a good- faith attempt at identifying a comparison group. The report acknowledges the 
potential for inequality between the groups. 
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VI. Review of the Validity of the Findings and Conclusions 

The report’s summary statistics appear accurate. Inferential statistics were used among the 
comparisons, which invite interpretations about relative performance between MPCP and 
MPS schools. However sufficient cautions were noted throughout the report in drawing 
inferences about relative effectiveness. 

VII. Usefulness of the Report for Guidance of Policy and Practice 

There is not much to take away from this report. The results are not particularly useful 
beyond providing a snapshot of how MPCP students and a comparison group of low- 
income MPS students perform on a battery of state exams. The report correctly cautions 
readers not to make causal inferences about the effects of either sector, given the 
descriptive rather than analytical nature of the analysis. 
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Notes and References 



1 The School Choice Demonstration Project (SCDP) is housed in the Department of Education Reform at the 
University of Arkansas (http:/ / www.uark.edu/ ua / der/ SCDP/ Research.html) . 

2 Jacob, A. M., &Wolf, P. J. (2012). Milwaukee Longitudinal School Choice Evaluation: Annual School Testing 
Summary Report 2010 - 11 . Fayetteville: University of Arkansas School Choice Demonstration Project. Retrieved 
February 27, 2012, from 

http:/ / www.uaedreform.org/ SCDP/ Milwaukee_ Research.html. 

3 “Very low performance” was defined as below a scale score of 400 in 4th grade math and reading and below a 
scale score of 450 in 8th and 10th grade math and reading (p. i). 

4 Eligibility for the federal lunch program is a family income at or below 185% of the poverty line. In contrast, 
initial eligibility for the MPCP is at an income ceiling of 175% of the poverty line, and renewal into the MPCP is at 
220 %. 
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